Integrating understandability in the evaluation of consumer health search engines

In this paper we propose a method that integrates the no- tion of understandability, as a factor of document relevance, into the evaluation of information retrieval systems for con- sumer health search. We consider the gain-discount evaluation framework (RBP, nDCG, ERR) and propose two understandability-based variants (uRBP) of rank biased precision, characterised by an estimation of understandability based on document readability and by different models of how readability influences user understanding of document content. The proposed uRBP measures are empirically contrasted to RBP by comparing system rankings obtained with each measure. The findings suggest that considering understandability along with topicality in the evaluation of in- formation retrieval systems lead to different claims about systems effectiveness than considering topicality alone

Software Tools for Indigenous Knowledge Management

Indigenous communities are beginning to realize the potential benefits which digital technologies can offer with regard to the documentation and preservation of their histories and cultures. However they are also coming to understand the opportunities for misuse and misappropriation of their knowledge which may accompany digitization. In this paper we describe a set of open source software tools which have been designed to enable indigenous communities to protect unique cultural knowledge and materials which have been preserved through digitization. The software tools described here enable authorized members of communities to: define and control the rights, accessibility and reuse of their digital resources; uphold traditional laws pertaining to secret/sacred knowledge or objects; prevent the misuse of indigenous heritage in culturally inappropriate or insensitive ways; ensure proper attribution to the traditional owners; and enable indigenous communities to describe their resources in their own words. Hopefully the deployment of such tools will contribute to the self-determination and empowerment of indigenous communities through the revitalization of their cultures and knowledge which have been eroded by colonization, western laws, western cultures and globalization

Payoffs and pitfalls in using knowledge‑bases for consumer health search

Consumer health search (CHS) is a challenging domain with vocabulary mismatch and considerable domain expertise hampering peoples’ ability to formulate effective queries. We posit that using knowledge bases for query reformulation may help alleviate this problem. How to exploit knowledge bases for effective CHS is nontrivial, involving a swathe of key choices and design decisions (many of which are not explored in the literature). Here we rigorously empirically evaluate the impact these different choices have on retrieval effectiveness. A state-of-the-art knowledge-base retrieval model—the Entity Query Feature Expansion model—was used to evaluate these choices, which include: which knowledge base to use (specialised vs. general purpose), how to construct the knowledge base, how to extract entities from queries and map them to entities in the knowledge base, what part of the knowledge base to use for query expansion, and if to augment the knowledge base search process with relevance feedback. While knowledge base retrieval has been proposed as a solution for CHS, this paper delves into the finer details of doing this effectively, highlighting both payoffs and pitfalls. It aims to provide some lessons to others in advancing the state-of-the-art in CHS

Collagen Fiber Re-Alignment and Uncrimping in Response to Loading: Determining Structure-Function Relationships Using a Developmental Tendon Mouse Model

Collagen fiber re-alignment and uncrimping are postulated mechanisms of structural response to load. It has been suggested that fibers re-orient in the direction of load and then uncrimp before collagen is tensioned and that in general, the structure is a result of the function tendons perform. However, little is known about how fiber re-alignment and uncrimping change in response to load, how this change relates to tendon mechanical properties, and if these changes are dependent on the underlying structure. Throughout postnatal development, dramatic structural and compositional changes occur in tendon. Postnatal tendons, with immature collagen networks, may respond to load in a different manner and timescale than mature collagen networks. Therefore, the overall objective of this study was to quantify the mechanical properties and structural response to load in a developmental mouse tendon model at 4, 10, 28 and 90 days old. Local collagen fiber re-alignment and crimp frequency were quantified throughout mechanical testing and local mechanical properties were measured. Throughout development, fiber re-alignment occurred at different points in the mechanical testing protocol. At early development, re-alignment was not identified until the linear (4 days) or toe-regions (10 days) of the mechanical test suggesting that fibers required a prolonged exposure to mechanical load before responding and that the immature collagen network present may delay re-alignment. The uncrimping of collagen fibers was identified during the toe-region of the mechanical test at all ages suggesting that crimp contributes to tendon nonlinear behavior. Additionally, results at 28 and 90 days suggested that collagen fiber crimp frequency decreased with increasing number of preconditioning cycles, which may affect toe-region properties. Mechanical properties and cross-sectional area increased throughout development. The insertion site demonstrated lower moduli values and a more disorganized fiber distribution compared to the midsubstance at all ages suggesting it experiences multi-axial loads. Further, the tendon locations demonstrated different re-alignment and crimp behaviors suggesting that locations may respond to load differently and develop at different rates. Results from this study suggest that structure affects the tendon\u27s ability to respond to load and that the loading protocol applied may affect the measurement of mechanical properties

QUT IElab at CLEF 2018 Consumer Health Search Task: Knowledge Base Retrieval for Consumer Health Search

In this paper we describe our participation to the CLEF 2018 Consumer Health Search Task, sub task IRTask1. This track aims to evaluate and advance search technologies aimed at supporting consumers to find health advice online. Our solution addressed this challenge by extending the Entity Query Feature Expansion model (EQFE), a knowledge base (KB) query expansion method. In previous work we showed that Wikipedia, UMLS and CHV can be effective as basis for CHS query expansions within the EQFE model. To obtain the query expansion terms, first, we mapped entity mentions to KB entities by performing exact matching. After mapping, we used the Title of the mapped KB entities as the source for expansion terms. For our first three expanded query sets, we expanded the original queries sourcing expansion terms from each of Wikipedia, the UMLS, and the CHV. For our fourth expanded query set, we combined expansion terms from Wikipedia and CHV

QUT ielab at CLEF 2017 e-Health IR Task: Knowledge Base Retrieval for Consumer Health Search

In this paper we describe our participation to the CLEF 2017 e-Health IR Task [6]. This track aims to evaluate and advance search technologies aimed at supporting consumers to and health advice online. Our solution addressed this challenge by developing a knowledge base (KB) query expansion method. We found that the two best KB query expansion methods are mapping entity mentions to KB entities by performing exact matching entity mentions to the KB aliases (EM-Aliases) and multi-matching entity mentions to all KB features (Title, Categories, Links, Aliases, and Body) (EM-All). After mapping between entity mentions to KB entities established, we found the Title of the mapped KB entities as the best source of expansion terms compared to the aliases or combination of both features. Finally, we also found that Relevance Feedback and Pseudo Relevance Feedback are effective to further improve the query effectiveness

Choices in Knowledge-Base Retrieval for Consumer Health Search

This paper investigates how retrieval using knowledge bases can be effectively translated to the consumer health search (CHS) domain. We posit that using knowledge bases for query reformulation may help to overcome some of the challenges in CHS. However, translating and implementing such approaches is nontrivial in CHS as it involves many design choices. We empirically evaluated the impact these different choices had on retrieval effectiveness. A state-of-the-art knowledge-base retrieval model—the Entity Query Feature Expansion model—was used to evaluate the following design choices: which knowledge base to use (specialised vs. generic), how to construct the knowledge base, how to extract entities from queries and map them to entities in the knowledge base, what part of the knowledge base to use for query expansion, and if to augment the KB search process with relevance feedback. While knowledge base retrieval has been proposed as a solution for CHS, this paper delves into the finer details of doing this effectively, highlighting both pitfalls and payoffs. It aims to provide some lessons to others in advancing the state-of-the-art in CHS

Towards Semantic Search and Inference in Electronic Medical Records

Background This paper presents a novel approach to searching electronic medical records that is based on concept matching rather than keyword matching. Aims The concept-based approach is intended to overcome specific challenges we identified in searching medical records. Method Queries and documents were transformed from their term-based originals into medical concepts as defined by the SNOMED-CT ontology. Results Evaluation on a real-world collection of medical records showed our concept-based approach outperformed a keyword baseline by 25% in Mean Average Precision. Conclusion The concept-based approach provides a framework for further development of inference based search systems for dealing with medical data

ChatGPT Hallucinates when Attributing Answers

Can ChatGPT provide evidence to support its answers? Does the evidence it suggests actually exist and does it really support its answer? We investigate these questions using a collection of domain-specific knowledge-based questions, specifically prompting ChatGPT to provide both an answer and supporting evidence in the form of references to external sources. We also investigate how different prompts impact answers and evidence. We find that ChatGPT provides correct or partially correct answers in about half of the cases (50.6% of the times), but its suggested references only exist 14% of the times. We further provide insights on the generated references that reveal common traits among the references that ChatGPT generates, and show how even if a reference provided by the model does exist, this reference often does not support the claims ChatGPT attributes to it. Our findings are important because (1) they are the first systematic analysis of the references created by ChatGPT in its answers; (2) they suggest that the model may leverage good quality information in producing correct answers, but is unable to attribute real evidence to support its answers. Prompts, raw result files and manual analysis are made publicly available

Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation

The current burnout rate of radiologists is high due to the large and ever growing number of Chest X-Rays (CXRs) needing interpretation and reporting. Promisingly, automatic CXR report generation has the potential to aid radiologists with this laborious task and improve patient care. Previous CXR report generation methods are limited by their diagnostic inaccuracy and their lack of alignment with the workflow of radiologists. To address these issues, we present a new method that utilises the longitudinal history available from a patient's previous CXR study when generating a report, which imitates a radiologist's workflow. We also propose a new reward for reinforcement learning based on CXR-BERT -- which captures the clinical semantic similarity between reports -- to further improve CXR report generation. We conduct experiments on the publicly available MIMIC-CXR dataset with metrics more closely correlated with radiologists' assessment of reporting. The results indicate capturing a patient's longitudinal history improves CXR report generation and that CXR-BERT is a promising alternative to the current state-of-the-art reward. Our approach generates radiology reports that are quantitatively more aligned with those of radiologists than previous methods while simultaneously offering a better pathway to clinical translation. Our Hugging Face checkpoint (https://huggingface.co/aehrc/cxrmate) and code (https://github.com/aehrc/cxrmate) are publicly available